Skeletonization: a Technique for Trimming the Fat from a Network via Relevance Assessment, 6.3.2 Other Pruning Methods

Stock performance modeling using neural networks: A comparative study with regression models', Neural Networks. fully connected network, it is not sufficient to stop removing weights when b P() first increases. This is particularly true given that both SBP and OBD are greedy algorithms, and they neglect interactions between multiple inputs and weights. Thus, removing additional inputs or weights even after b P() first increases can lead to a further reduction of the prediction risk and thus yield a smaller final network. A number of other pruning methods besides those we have described here are potentially effective and should be considered when constructing neural network models. These include the irrelevant hidden unit and irrelevant input hypothesis tests (White, 1989), pruning of units via skeletonization (Mozer and Smolensky, 1990), optimal brain surgeon OBS (Hassibi and Stork, 1993), and principal components pruning PCP (Levin et al., 1994). It is important to note that all these methods, along with OBD and our method of input pruning via SBP, are closely related to the Wald hypothesis testing procedure (see for example Buse (1982)). In fact, the saliencies used in OBD, OBS, and SBP are special cases of the Wald test statistic. Acknowledgements The author thanks Joachim Utans for help in preparing this manuscript and Steve Rehfuss and Eric Wan for careful proof readings. The author is responsible for any remaining errors.Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation', Numer. OBD is designed to select those weights in the network whose removal will have a small effect on the training average squared error (ASE). Assuming that the original network is too large, removing these weights and retraining the now smaller network should improve the generalization performance.